MSQI IN ACTION: Findings from the 2014-15 Evaluation 


I. BACKGROUND 


In 201 1 , the NYCDOE and the City Council's Middle School Taskforce launched the Middle 
School Quality Initiative (MSQI) as an instructional intervention for improving literacy levels in 
the lowest performing middle schools in New York City. The initiative grew rapidly and now 
serves over 100 schools and 53,000 students (table 1). 


Cohorts (Year of Entry) 

# of Schools 

# of Students 

# of Teachers 

Cohort 1 (2011) 

21 

9,284 

782 

Cohort 2 (2012) 

19 

13,762 

1,047 

Cohort 3 (2013) 

17 

10,296 

719 

Cohort 3 Extra (2013) 

19 

8,280 

601 

Cohort 4 (2015) 

31 

12,245 

838 

Schools discontinued 

11 

5,503 

444 

Active schools in SY 2015-16 

107 

53,867 

3,987 



MSQI 


When designing last year's evaluation, MSQI leadership and the Research and Policy Support 
Group (RPSG) agreed to focus on uniformly collecting implementation fidelity data. The 2014- 
2015 fidelity survey provides a snapshot of MSQI implementation in SY 2014-15, describing 
pillar activities across all 87 schools. 


II. EVALUATION FINDINGS 


Findings on the four key evaluation questions for SY 2014-15 are presented below. 

o What are the patterns of implementation fidelity across MSQI schools? 

From MSQI's initiation, school leaders possessed ample freedom to implement the most relevant 
pillars for their schools. Few schools were able to simultaneously take all pillar activities from an early 
stage of development (pre-emergent or emergent) to a deeper level of engagement (active or full). 

• Schools need support to improve their proctoring of universal literacy screeners (DRP) and the 
subsequent use of results to maximize the benefit of these exams for students. 

• Strategic Reading Periods are in progress, though few schools have yet to implement it as 
MSQI recommends (5 days/week, 40 minutes per period, using tiered interventions). 

• Few schools are monitoring essential data (ELA, DRP, and secondary diagnostics) in a single 
data system, or discussing student scores during teacher team meetings. 

• Promoted by multiple DOE initiatives, most schools report addressing literacy across content 
areas, holding teacher team meetings, and planning professional learning opportunities. 

© What school-level contextual factors (including principal leadership capacity, teacher 
performance, and student ELA outcomes) are associated with MSQI implementation fidelity? 

• Schools with both effective principals and high ELA performance exhibit high fidelity. 

• Coaches mitigate the difficulty of implementing MSQI in schools with low ELA performance. 

• Coaches mitigate the difficulty of implementing MSQI in schools with a low proportion of 
effective teachers. 

• There is no difference in fidelity between MSQI cohorts. 

© What is the relationship, if any, among MSQI implementation fidelity and gains in literacy 
achievement as measured through the DRP, net of contextual factors? 

• Schools with higher ratings for rigorous instruction, as indicated by the NYC School Survey, tend 
to show greater DRP growth. 

• Schools with low ELA scores are just as likely as schools with high ELA scores to show DRP growth. 

• There is uncertainty about the relationship between fidelity and DRP growth. 
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Q What additional school-level factors may influence MSQI implementation? 

• School-Wide Literacy Vision: Of the 86 schools interviewed, 56 principals provided a neutral or 
poorly-articulated literacy vision for their schools. A common difficulty was a focus on very distant 
outcomes, such as "college and career readiness", to the exclusion of more proximate outcomes. 

• School Libraries: A key factor in implementing a successful literacy program is to ensure that 
students have access to reading materials throughout the school day. Close to 50% report that 
their library is staffed less than part-time and only 45% report that students have regular access to 
materials in library. 

• Program Coherence: Schools often implement some pillars with higher fidelity than others. 

This suggests that schools may not be approaching these programmatic activities in a cohesive 
manner, providing equal attention to all 5 pillars as supporting one another to achieve the goal of 
all eighth graders reading on or above grade-level. 


III. ADDITIONAL FINDINGS 


• Respondents expressed a desire for greater inter-school collaboration. "Sustainability of MSQI 
activities is not a money issue, it's an issue of networks. It would be difficult to continue without 
having access to a support network and/or community to bounce ideas off each other." 

• Schools requested stronger alignment across MSQI, CCLS literacy requirements, and curricula for 
content areas in order to make integration of MSQI more feasible. 

• There is a concern about over testing in schools, suggesting that numerous assessments are 
strenuous for students and for teachers. "There is just an overwhelming number of tests and the 
testing anxiety that students experience reduces their focus on non-DOE mandated, low stakes 
test like the DRP. " 

• Findings suggest that schools have misunderstood SRPs as defined by MSQI (40 minute period, 

5 days per week, with tiered literacy instruction, and homogenous grouping). Limited time in 
the school day, staff unavailability, teacher contract issues, and school size make accurate SRP 
implementation difficult. 


IV. RECOMMENDATIONS 


Q Provide literacy coaching in all MSQI schools. 

Findings suggest that schools with low ELA performance tend to have low fidelity, but that 
literacy coaches may mitigate this difficulty. Maintaining a ratio of 1 coach to 6-8 schools is 
critical for achieving such mitigation. 

© Build upon programs aimed at addressing contextual factors and the Framework for 
Great Schools. 

Schools with a high proportion of effective teachers, and high ratings on instructional rigor tend 
to show greater DRP growth. The evaluation also found that schools with stronger leadership 
tend to have higher fidelity. Together, these findings suggest that assisting schools to improve 
these broader contextual factors, specifically strengthening school leadership and providing 
literacy coaching support, influences program implementation. 

© Support school efforts to deepen the understanding and implementation of MSQI's 5 pillars. 

The evaluation indicates that schools need targeted guidance to deepen their work with the 
pillar activities and achieve full fidelity. It may be valuable for literacy coaches to 
concentrate on the details of implementing activities jointly emphasized by multiple DOE 
initiatives mentioned above. 

O Support principals in articulating a literacy vision and establishing program coherence. 

While it makes sense to let schools adopt the different parts of MSQI according to what fits 
them best, the pillars support one another and are theorized to lead to student learning gains 
best when implemented together. Schools should be encouraged to approach programmatic 
activities in a cohesive manner. Additionally, only one third of school leaders interviewed were 
able to share a we 1 1 -articulated vision for literacy in their schools. Helping school leaders build 
and articulate a literacy vision is a key priority for MSQI literacy coaches. 



All of our scholars will 
graduate from middle 
school reading at (or 
above) grade level. This 
work begins with our 
teachers, who will as a 
community learn the 
science behind literacy 
development and 
enhance their capacity to 
implement high-impact 
literacy instruction in 
their classrooms. In all 
of our classes, we will 
plan opportunities for 
students to engage with 
a variety of challenging 
texts, and students 
will have meaningful 
discussions to help 
them strengthen their 
comprehension and 
appreciation of the 
written word. Most 
importantly, we will 
celebrate books and 
reading, pushing 
students to understand 
the relationship between 
literacy and personal 
and academic success. 

~ Literacy vision from 
an MSQI principal 




